Methods and systems for adapting a user environment

ABSTRACT

A method of adapting an environment of a terminal ( 2 ) includes: —receiving data ( 27 ) from an external source at the terminal ( 2 ); —receiving monitoring data from a system for monitoring at least one user in the environment; and —controlling at least one output device for providing an output perceptible in the environment, the output being adjusted in dependence on the monitoring data. At least one segment including data pertaining to at least one mental state is retrieved from the data ( 27 ) from the external source, and the adjustment in dependence on the monitoring data is effected at least partly on the basis of the data pertaining to the at least one mental state.

FIELD OF THE INVENTION

The invention relates to a method of adapting an environment of aterminal, including:

receiving monitoring data from a system for monitoring at least one userin the environment.

The invention also relates to a system for adapting an environment of aterminal, including:

an interface for receiving monitoring data from a system for monitoringat least one user in the environment.

The invention also relates to a method of influencing an environment ofa terminal, including:

the terminal being arranged to receive monitoring data from a system formonitoring at least one user in the environment.

The invention also relates to a system for influencing an environment ofa terminal, configured to provide data for delivery to the terminal,

the terminal being arranged to receive monitoring data from a system formonitoring at least one user in the environment.

The invention also relates to a signal, and a computer program.

BACKGROUND OF THE INVENTION

US 2005-0223237 A1 discloses an emotion-controlled system for processingmultimedia data. It comprises a multimedia system for presentingmultimedia content to the user, an emotion model means for determiningthe emotional state of the user during the presentation of themulti-media content and an editing unit for changing the multimediacontent in accordance with the emotional state of the user in order topresent the changed multimedia content by the multimedia system. Themultimedia system may comprise not only display and loudspeakers butalso other environmental devices such as lights or the like. Accordingto the user's emotional state, a home entertainment system might receiveand process emotion information and adapt the environment to suchstimuli. If, for example, the user is watching a scary movie and thephysical changes related to fear, such as a high heart beat rate or highblood pressure are submitted to the emotion model means and thenafterwards the emotional state of fear is submitted to the editingmeans, then the multimedia system may adapt the environment by dimmingdown the light or increasing the sound in order to emphasize the emotionfelt by the user.

A problem of the known system is that the user might not be particularlyscared by the movie on its own, so that the environment is never adaptedto emphasize such an emotion.

SUMMARY OF THE INVENTION

It is an object of the invention to provide methods, systems, a signaland a computer program of the types mentioned above, that allow theprovider of content such as entertainment material to provide a userwith an immersive experience with relatively few incongruencies.

This object is achieved by the method according to the invention, whichincludes:

receiving data from an external source at the terminal;

receiving monitoring data from a system for monitoring at least one userin the environment; and

controlling at least one output device for providing an outputperceptible in the environment, the output being adjusted in dependenceon the monitoring data,

wherein at least one segment including data pertaining to at least onemental state is retrieved from the data from the external source, andthe adjustment in dependence on the monitoring data is effected at leastpartly on the basis of the data pertaining to the at least one mentalstate.

Because monitoring data is received from a system for monitoring atleast one user in the environment and at least one output device forproviding an output perceptible in the environment is controlled suchthat the output is adjusted in dependence on the monitoring data, a formof closed-loop control can be implemented at the terminal. Because theadjustment is effected in dependence on the monitoring data at leastpartly on the basis of data pertaining to the at least one mental state,the closed-loop control is effected such as to arrive at a target mentalstate. Because the terminal receives data and retrieves from the data atleast one segment including the data pertaining to at least one mentalstate the target mental state is provided externally. In particular, itcan be provided by an author of content data. Thus, the terminal adaptsthe outputs to achieve a target mental state that the author hasinstructed it to. A remote author can thus provide an appropriateimmersive experience.

In an embodiment, the data from an external source comprises a datastream.

This embodiment has the effect that the target mental state can varyover time, as the data stream is received and parsed at the terminal.

In an embodiment, the data pertaining to at least one mental stateincludes data carrying information from which the dependency between themonitoring data and the output is at least partly derivable.

An effect of this feature is that the terminal is provided with theinformation on the control algorithm to implement to achieve a targetmental state based on the monitoring data. Thus, the terminal need notbe provided with an elaborate model of a user. More in particular, theterminal need not be provided with models and algorithms for achievingevery one of a wide range of emotional states. Instead, it is providedby an external source with that data needed to achieve the targetemotional state intended by the remote author of the immersive effects.This allows the use of a terminal with at least less memory. Moreover,it means that the author need not take account of different models thatmight otherwise be present in different types of terminals. Also, theexternal author can incorporate the latest insights into how toinfluence moods and what relations exist between moods and physiologicalor behavioral phenomena in the data provided by him. Thus, there is lessneed to rely on separate updates of such models and relations programmedin terminals already in use.

In an embodiment, the data pertaining to at least one mental stateincludes data representative of a boundary value of an allowable rangeof values of at least one signal obtainable from the monitoring data,and the adjustment of the output is carried out such as to keep valuesof the signal obtainable from the monitoring data within the allowablerange of values.

This feature allows one to calibrate terminals, in the sense that theterminal can never implement a model of the user or of the relationbetween mental state and physiological or behavioral symptoms thatresults in excessive values of parameters characterizing such symptoms.

In an embodiment, segments including data pertaining to at least onemental state are received in synchrony with content data for renderingin at least one of a visible and an audible manner at the terminal, andthe adjustment of the output perceptible in the environment issynchronized with a rendering of the content data at the terminal.

This embodiment allows an author of content data for rendering in atleast one of a visible and an audible manner to provide an accompanyingstream of data for adapting the viewing or listening environment toachieve a mental state of the viewer or listener that is appropriate tothe content. More particularly, this mental state can change as theaudio and video progress, e.g. the feelings engendered by the film'splot change from a state of suspense to a state of relief.

An embodiment includes receiving at least one segment of the data fromthe external source including data representative of aspects of theperceptible output to be adjusted, and modifying the data representativeof aspects of the perceptible output to be adjusted based on the datapertaining to the at least one mental state in dependence on themonitoring data.

This embodiment allows an author to specify how a particular mentalstate is to be arrived at. For example, the author can specify that ascared state is to be achieved by adjustment of air currents generatedby a fan or by adjustment of haptic effects generated by a forcefeedback device. In particular where the data stream is provided toaccompany audiovisual content to be rendered, this can be useful in thatthe scared state might be preferably achieved using the former modalitywhen the video depicts a windy scene or using the latter modality whenthe video depicts a marching army or the like.

In a variant of this embodiment, the data representative of aspects ofthe perceptible output to be adjusted is further modified on the basisof data representative of capabilities of at least one of the terminaland the output devices controllable by the terminal.

This variant ensures that the data stream can be handled by a relativelywide range of terminals and output devices controlled by such terminals.The data representative of capabilities of at least one of the terminaland the output devices controllable by the terminal may include rulesfor replacement of instructions contained in the data representative ofaspects of the perceptible output to be adjusted or it may lead tocertain instructions being ignored.

An embodiment of the method includes receiving data from an externalsource including both segments including the data representative ofaspects of the perceptible output to be adjusted and segments includingthe data pertaining to the at least one mental state, and separating therespective segments from each other.

This embodiment further widens the range of terminals and decoderscomprised in such terminals that can handle the data stream, in thatthose terminals that do not receive monitoring data can still controloutput devices. In that case, they would use only the datarepresentative of aspects of the perceptible output to be adjusted.Moreover, the separation of the data allows for the data segments to beprovided in simpler formats than would be the case if a single datacontainer were to indicate both the emotional state to be achieved—andhow—and the output modalities to be used. Such a container would need tocome with elaborate case-statements, which would be relatively difficultfor a simple decoder to parse.

In a variant, the data from the external source comprises a data stream,and the separation is effected by demultiplexing the data stream toretrieve separately identifiable data packets including the datarepresentative of aspects of the perceptible output to be adjusted anddata packets including the data pertaining to the at least one mentalstate.

This variant has the effect that synchronization of the segments isrelatively easy to achieve. Moreover, especially where the data streamsare transmitted, e.g. broadcast or multicast, over a network, theretrieval of the segments at the terminal is made easier.

In a further method of adapting an environment of a terminal thatincludes:

receiving data from an external source at the terminal;

receiving monitoring data from a system for monitoring at least one userin the environment; and

controlling at least one output device for providing an outputperceptible in the environment, the output being adjusted in dependenceon the monitoring data, wherein the at least one segment including datapertaining to at least one mental state is retrieved from the data fromthe external source, and the adjustment is effected in dependence on themonitoring data at least partly on the basis of the data pertaining tothe at least one mental state, a closed-loop control system is used. Afeedback loop of the control system includes a model for classifying amental state of at least one user based on the monitoring data. Themental state is classified as a set of values in N-dimensional space,wherein N≧1.

This has the effect that a suitable control variable is provided. Onecan derive an error signal by subtracting the model output from thetarget value.

In a variant of this embodiment, the adjusted output includes arendering of at least one of audio, video, image, tactile feedback,olfactory and gustatory data, the adjustment including selection of thedata to be rendered as perceptible output from a collection of availabledata.

In this variant, the mood of a user can be influenced comprehensivelythrough selection of all perceptible outputs. Moreover, the collectionof available data can be provided in advance.

Where the data to be rendered is selected, the control system caninclude a discrete controller.

Thus, discrete outputs that remain selected for a while are provided, inorder to give the user time to take in the changed output. Especially inthe case of cognitive data, the monitoring data will not show a veryquick response to changed output.

Alternatively, where output is adjusted in dependence on the monitoringdata by changing settings of an output device regulating how output isrendered in perceptible form, the control system can comprise acontinuous controller.

Thus, for example, the volume of sound reproduction can be controlledcontinuously. It will thus not remain too loud for a longer time, but beturned down almost as soon as an inference has been made that the userhas become too agitated, for example.

According to another aspect, the system for adapting an environment of aterminal according to the invention includes:

an interface for receiving data from an external source at the terminal;

an interface for receiving monitoring data from a system for monitoringat least one user in the environment; and

a device for controlling at least one output device for providing anoutput perceptible in the environment,

wherein the system is configured to cause the output to be adjusted independence on the monitoring data, and the system is configured toretrieve from the data from the external source at least one segmentincluding data pertaining to at least one mental state, and to effectthe adjustment in dependence on the monitoring data on the basis of thedata pertaining to the at least one mental state.

In an embodiment, the system is configured to carry out a methodaccording to the invention.

According to another aspect, the method of influencing an environment ofa terminal according to the invention includes:

providing data to the terminal,

the terminal being arranged to receive monitoring data from a system formonitoring at least one user in the environment,

wherein the terminal is arranged to control at least one output devicefor providing an output perceptible in the environment, the output beingadjusted in dependence on the monitoring data, the method furtherincluding providing data including at least one segment including datapertaining to a mental state to a terminal arranged to effect theadjustment in dependence on the monitoring data at least partly on thebasis of the data pertaining to the mental state.

The method is suitable for implementation in a content authoring system,e.g. for providing data for broadcast or multicast. Alternatively, sucha data stream can be provided on a data carrier such as a DVD or CD, inparticular in combination with audiovisual data.

In an embodiment of the method, the data pertaining to at least onemental state includes data carrying information from which thedependency between the monitoring data and the output is at least partlyderivable.

Thus, the terminal need be configured primarily to decode the data, butit need not previously have been programmed with every possibledependency between every conceivable mental state and the correspondingphysiological and behavioral symptoms of users in such states.

In an embodiment of the method of influencing an environment of aterminal, the data pertaining to at least one mental state includes datarepresentative of a boundary value of an allowable range of values of atleast one signal obtainable from the monitoring data.

This data allows the entity carrying out the method to maximize orminimize certain monitored physiological or behavioral variables. Theterminal provided with the data stream can thus be calibrated to ensurethat any adjustments to its output that result in a change of mentalstate of a user remain within acceptable boundaries.

In an embodiment of the method of influencing an environment of aterminal, segments including data pertaining to at least one mentalstate are provided in synchrony with content data for rendering in atleast one of a visible and an audible manner at the terminal.

Thus, the entity carrying out the method can provide an immersiveexperience at the location where the audiovisual content data isrendered.

In an embodiment of the method of influencing an environment of aterminal, at least one segment of data including data representative ofaspects of the perceptible output to be adjusted is provided.

Thus, the entity carrying out the method can also influence the modalityused to change the mental state of the persons in the environment of theterminal.

In a variant, the segments including the data representative of aspectsof the perceptible output to be adjusted and the segments including thedata pertaining to the at least one mental state are provided in anidentifiably separate manner.

This allows a terminal to retrieve only one of the two types ofsegments. In particular, if a legacy terminal has no capability ofprocessing monitoring data representative of physiological or behavioralsymptoms of at least one user in the environment of the terminal, thenonly the data representative of the perceptible output to be adjustedcan be retrieved and parsed.

According to another aspect, the system for influencing an environmentof a terminal according to the invention is configured to provide datafor delivery to the terminal,

the terminal being arranged to receive monitoring data from a system formonitoring at least one user in the environment,

wherein the terminal is further arranged to control at least one outputdevice for providing an output perceptible in the environment, theoutput being adjusted in dependence on the monitoring data, and whereinthe system is configured to provide data including at least one segmentincluding data on the basis of which the terminal is able to adjust theoutput in dependence on the monitoring data.

According to another aspect, the signal according to the inventioncarries a data stream including at least one segment including datapertaining to a mental state, which data includes information on thebasis of which a terminal arranged to control at least one output devicefor providing an output perceptible in an environment of the terminal isable to adjust the output in dependence on monitoring data provided by asystem for monitoring at least one user in an environment of theterminal.

According to another aspect, the computer program according to theinvention includes a set of instructions capable, when incorporated in amachine-readable medium, of causing a system having informationprocessing capabilities to perform a method according to the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be explained in further detail with reference tothe accompanying drawings, in which:

FIG. 1 is a diagram providing an overview of a system of distributingdata for adapting an environment of a receiver decoder;

FIG. 2 is a block diagram of a decoder model for decoding the data foradapting the environment of the receiver decoder of FIG. 1;

FIG. 3 is a schematic illustration of a packet-based data streamcarrying the data for adapting the environment of the receiver decoderof FIG. 1; and

FIG. 4 is a block diagram describing in functional terms a furtherembodiment of a system for adapting the environment of a terminal.

DETAILED DESCRIPTION

FIG. 1 illustrates in a schematic way an architecture for distributingdata. In the following, the data will be taken to include at least oneof audio, video and image data. The content data is provided incombination with time-varying emotion data. In certain embodiments, theemotion data is synchronized with the content data so that differentmental states can be engendered in the persons accessing the contentdata at the appropriate times.

To provide the emotion data, an authoring system 1 is provided. Theauthoring system 1 is an example of a system for influencing anenvironment of a terminal 2 at which the content data is rendered inperceptible form. The terminal 2 can be implemented as a set-top box,personal computer, handheld computer, cellular telephone handset,digital television, etc. In one embodiment, the output of the authoringsystem 1 is provided on a data carrier 3, e.g. an optical disk, directlyto the terminal 2. In another embodiment, it is provided on a datacarrier 4 or via a network link to a server 5, which may be comprised ina broadcast network head-end, for distribution across a network 6 to theterminal 2 and other similar terminals. The network 6 can comprise acellular network, a long-distance optical network, a satellite or cabletelevision network, etc. Distribution may be in the form of a broadcast,multicast or direct transmission (e.g. download on demand to theterminal 2). In the illustrated embodiment, the content data isdistributed in a digital television format, e.g. DVB or ATSC.

The terminal 2 receives the content data with the emotion data via anetwork interface 7, tuner 8, demodulator 9 and demultiplexer 10 in caseof distribution via the network 6. The illustrated terminal 2 alsocomprises a read unit 11 for reading the contents of the data carrier 3.

In a known manner, the terminal 2 comprises a video codec 12, anddisplay driver 13 for rendering the video component of a selected streamof content data on a display 14. An audio codec 15 and audio outputstage 16 provides signals to loudspeakers 17,18.

The terminal 2 operates under the control of a processor 19 and mainmemory 20 for executing software, including instructions for parsingemotions data and implementing a decoder, a model of which is shown inFIG. 2. In addition, the terminal 2 is comprised in an ambient systemfurther including ambient output devices 21,22 and a driver 23 therefor.The ambient output devices 21,22 are representative of a wide class ofoutput devices for rendering perceptible output other than audio, videoand image data. Examples include lighting devices for ambient lighting,in particular those arranged to provided directed light of adjustablecolor, intensity and direction, devices for influencing the ambientclimate, e.g. temperature, humidity, and devices for providing tactilefeedback, e.g. rumblers. The ambient output devices 21,22 can be worn ona user's person, e.g. comprised in a wrist-band, glove or head-set.

The terminal 2 also includes an interface 24 to monitoring devices25,26, arranged to enable the terminal 2 to obtain monitoring datarepresentative of at least one of physiological and behavioral symptomsexhibited by respective users of the terminal 2. Alternatively oradditionally, the monitoring data may characterize the userscollectively. The monitoring data includes data characterizing at leastone of physiological and behavioral phenomena. Examples of the formerinclude blood pressure, skin temperature, skin conductivity, heart rate,respiration rate and changes to these variables. Examples of the latterare generally characteristics of movement of the user or users, e.g.obtained by interpreting images taken with a digital camera. Forexample, facial expressions, the frequency of movement, the gazedirection and similar characteristics can be evaluated.

The monitoring data allows the terminal 2 to implement a form ofclosed-loop control of the mental state of the user or the users in theenvironment of the terminal 2. Thus, the mental state of the user isinfluenced to achieve a target mental state in accordance with theemotion data generated remotely on the authoring system 1.

FIG. 2 illustrates a data stream 27 in accordance with internationalstandard ISO/IEC 13818-1 (MPEG-2 Systems) as an example of a data streamfor carrying emotion data. The data stream 27 is comprised of transportstream packets 28 a-d, each comprising a header 29 a-d and a payload 30a-d of a fixed data length. The transport stream packets 28 a-d carryProgram Elementary Stream (PES) packets 31 a-c, each also comprising aheader 32 a-c and a payload 33.

In the cases in which the emotion data is provided in synchrony withaudio, video or audiovisual content data, the emotion data is part of aprogram. A program comprises a plurality of elementary streams. Eachelementary stream is comprised of PES packets 31 a-c which areseparately identifiable by means of a stream identifier 34 in the PESpacket header 32 a. The PES packet header 32 a includes further fields,which are not described here, except for a field 35 including a timestamp for determining at least one of a decode and presentation timerelative to a clock reference carried in the data stream 27. Thus,synchronized rendering of the program components carried by the datastream 27 can be achieved.

Because any TS packet 28 carries data of one and only one elementarystream, a separate Packet Identifier (PID) in the TS packet header 29also identifies the various elementary streams making up a program. Timestamps for synchronizing the elementary streams can in fact be includedin the TS packet headers 29 a-d, which in any case carry the clockreference.

As is described more fully in international standard ISO/IEC 13818-1,any data stream 27 made up of TS packets 28 a-d comprises packets with afixed identifier that carry data representative of a Program AssociationTable (PAT), not shown. This table identifies the available programs inthe data stream 27, and points to the packets carrying a program maptable (PMT). There is one such PMT for each program. The PMT identifiesthe packets carrying the different elementary streams, and their type.Depending on the embodiment, the PMT includes the PID of the TS packets28 a-d or the elementary stream identifiers of the PES packets 31 a-c.

In the example used herein, the emotion data are carried in at least oneseparate elementary stream, each having its own respective identifiervalue, indicated in the PMT of the program to which it belongs. Wherethe emotion data is provided in synchrony with audio and video data, thePMT will identify the stream or streams carrying the audio data, thestream or streams carrying the video data and the stream or streamscarrying the emotion data. The type of the stream or streams carryingemotion data will be “private”, so that legacy terminals may ignorethese streams. Thus, whereas in the terminal 2 of FIG. 1, thedemultiplexer 10 is set to retrieve the emotion data, a legacy terminalwould ignore them, and retrieve only those carrying the audio and videodata.

FIG. 3 shows a decoder for a first embodiment of the terminal 2 for usein decoding emotion data carried in the data stream 27 and adjusting theoutput of at least the ambient output devices 21,22. The decoder isillustrated in terms of functional components, which may be implementedin a combination of hardware and software components in any of variousways.

The input to the decoder comprises the data stream 27 of FIG. 2. Apacket filter 36 takes out only those TS packets 28 a-d that comprisepacket headers 29 a-d with a certain PID value. These are the TS packets28 a-d carrying the emotion data. The packet filter 36 knows which PIDvalues to look for from the PMT, which has been parsed previously, andwhich lists the PID values and associated descriptions of the elementarystreams that form a program as explained above. The packet filter 36 istypically comprised in the demultiplexer 10 (FIG. 1).

In an embodiment of the method appropriate to the decoder model of FIG.2, there are two types of data objects comprised in the emotion data,namely sensory scene description objects and emotion compositionobjects. In the following, the terms “object” and “segment” will be usedinterchangeably, segment being the term used to refer to objectsembedded in a data stream.

The emotion composition objects comprise data pertaining to at least onemental state. This information informs the terminal of the target mentalstate to be achieved in the persons in the environment of the terminal2.

The sensory scene description objects include data representative ofaspects of the perceptible output of the ambient output devices 21,22and, optionally, the display system and sound system comprised in theterminal and display 14 and loudspeakers 17,18, that are to be adjustedin order to influence the mental state of the users. That is to say thatthe sensory scene description object will specify whether a particularmental state is to be achieved by controlling the output of a tactilefeedback device, the color temperature of lighting units, the soundlevel of the sound system, etc. The actual mental state to be achievedand the adjustments to be made to the selected output devices to achievethe mental state are specified in the emotion composition objects.

In the illustrated embodiment, the emotion composition objects andsensory scene description objects are identifiable as separate objectswithin a single elementary stream. In an alternative embodiment, theemotion composition objects and the sensory scene description objectsare comprised in separate segments, each carried in a differentrespective elementary stream identifiable by a different respective PIDvalue. Thus, a legacy decoder could ignore the emotion compositionobjects, merely using the sensory scene description objects to adjustthe output of the ambient output devices 21,22. In that case, thesensory scene description objects would also include the level of thevarious types of output specified by them, e.g. the strength of theoutput of a fan or tactile feedback device, the color and intensity oflight provided by ambient lighting devices etc. However, such a legacydecoder would not take into account the monitoring data obtained fromthe monitoring devices 25,26 when adjusting the output of the ambientsystem.

Returning to the decoder model, the TS packets 28 a-d retrieved by thepacket filter 36 are placed in a transport buffer 37. There, the packetheaders 29 a-d,32 a-c are removed. The decoding time stamps andpresentation time stamps are removed and associated with the emotionstream segments (sensory scene description segments and emotioncomposition segments). The segments are then placed in a coded databuffer 38.

An emotion stream processor 39 decodes and analyses the various segmentsof the emotion stream. Emotion composition segments are placed in anemotion command buffer 40 and sensory scene description segments areplaced in a decoded sensory scene description object buffer 41.

As mentioned, the sensory scene description segments carry the data thatdescribe how to change the ambience of the terminal 2, such as whetherto change the lighting conditions, air flow, sound, smell, temperatureor whether to provide tactile, audible or visible feedback. This data isgenerally encoded in a markup language. Examples of suitable dataformats include MPEG-V, MPEG-RoSE or Ambi-X.

Emotion composition segments include any of three types of data, namelydata describing a target mental state, emotion window definition dataand interactive emotion composition data. The data will generally be ina markup language. It may include data in the W3C emotions markuplanguage or MPEG-V format, extended to provide the functionality of thethree types of emotion composition segment.

The emotion window definition data describes boundaries of the emotionalstate of the user. It is used as a calibration step, so that the user isnot confronted with an undesired extreme sensory experience. Thus, thisdata includes data representative of at least one boundary value of anallowable range of values of at least one signal obtainable from themonitoring data received through the interface 24. The terminal 2adjusts the output of the ambient output devices 21,22 so as to keepvalues of the signal obtainable from the monitoring data within theallowable range of values. In one embodiment, the signal obtainable fromthe monitoring data that is kept within an allowable range is a signalcorresponding to a value of a variable characterizing the physiologicalor behavioral phenomena (blood pressure, heart rate, a measure ofactivity such as fidgeting, etc.). In another embodiment, the signal isa signal obtained by applying a model of a user to such a signal, inorder to obtain a value or values of a variable corresponding to adimension of a user's mental state (degree of arousal, degree ofemotionality, degree of alertness, etc.).

The interactive emotion composition segments define how to react to achange in the measured emotional state of the user or collective ofusers. For example, it may define that if a user is feeling tense, theambient lighting should change color. Indeed, it may define a functionthat relates the change in color to a change in a variablecharacterizing the tenseness of the user. Thus, the terminal 2 canderive a dependency between at least a sub-set of the monitoring dataand the output to be provided by the ambient output devices 21,22.

Returning to the decoder model of FIG. 3, the signals from themonitoring devices 25,26 are processed by a monitoring signal processor42 to obtain a signal for a sensory scene composition controller 43. Thesensory scene composition controller uses the instructions in thesensory scene composition segments on how to respond to signals that arefed in in real-time from the monitoring signal processor 42. The sensoryscene composition controller 43, based on the instructions in theemotion composition segments and the monitoring data, changes thedecoded sensory scene description objects contained in the decodedsensory scene description object buffer 41. At this stage, it alsoensures that the instructions in the emotion window definition data areobserved.

The decoded sensory scene description objects that have been adjusted inthis way move to a sensor scene description buffer 45. In the process, asensory device capabilities mask 44 is applied. This mask 44 comprisesdata representative of capabilities of at least the terminal 2 or theoutput devices 12-18,21,22, controllable by the terminal 2, or both. Theapplication of this mask 44 occurs at the final stage of the decodingprocess. Thus, the sensory scene description in the sensory scenedescription buffer 45, which contains the commands required to drive theoutput devices 12-18,21,22, always matches the capabilities of theavailable sensory rendering platform.

FIG. 4 shows an alternative model of a system for adapting theenvironment of the terminal 2, also intended for implementation in theterminal 2. In this embodiment, the terminal 2 receives data pertainingto at least one mental state that includes at least the target mentalstate and, optionally, the manner in which the target mental state is tobe achieved, e.g. in the form also of sensory scene description objectsthat describe which outputs to adjust and emotion scene compositionobjects that provide a model of a user in the form of functionsdescribing how the mental state depends on monitoring data.

The system includes a controller 46, to be described in more detailfurther below. It also includes an actuation system 47 for providingoutput perceptible in the environment of the terminal 2. The actuationsystem 47 is embodied in the ambient output devices 21,22, in theprocessor 19, video codec 12, display driver 13 and display 14, and inthe processor 19, audio codec 15, audio output stage 16, andloudspeakers 17,18, or in a sub-set of these devices, depending on theembodiment. They provide stimuli to which a human body 48 reacts. Inparticular, they provide stimuli of any one or more of the five senses(sight, smell, hearing, touch and taste).

A measurement system 49 provides monitoring data to aclassifier/interpreter 50 representing a model of the human body 48. Theclassifier/interpreter 50 provides data representative of a mental stateas output. This data is in the form of a set of values in N-dimensionalspace, wherein N≧1. Thus, it can be subtracted from the target mentalstate, which is in the same format. The difference signal is the inputto the controller 46. The number N of dimensions can range from one tofifteen, in particular from one to three. Generally, there will be twodimensions, e.g. the valence and arousal dimensions of one widely usedmodel of a person's emotional state. Higher values are useful when moreintricate emotion models are used, for instance appraisal-based emotiondescriptions.

In one embodiment, the terminal 2 is used to display images on thedisplay 14. The perceptible output in this case is adjusted by selectionof the source data for display (the picture). In this embodiment, thecontroller 46 is a discrete controller. In a similar embodiment, theterminal 2 selects scenes, switching between several alternativevariants of a particular program. For example, there may be severalelementary streams carrying video data, wherein the elementary streamseach comprise a different variant of one story but different detailsand/or video settings. In a variant, the story truly forks and joins asingle story line, in which case several audio streams with differentdialogues are also provided. Alternatively, there can be one videostream but different audio streams, e.g. each with a different style ofbackground music, with the controller 46 functioning to select theappropriate stream in dependence on the error signal.

In another embodiment, an overlay is provided on the display 14, theoverlay covering only part of the area available for display of othercontent. In particular, an emoticon can be selected for display. In thisembodiment, the controller 46 will generally be implemented in the formof a bang-bang controller. A bang-bang controller is a controller thatswitches between states, and has hysteresis.

In another embodiment, the controller 46 is a continuous controller,e.g. a PID, ID, P, D or I controller, which governs the settings of theoutput devices used to render pre-selected content or generally adjustthe ambience of the terminal 2. Assuming that the terminal 2 is providedwith a data stream 27 with embedded mental state targets, the terminal 2will gradually adjust the immersiveness of the audiovisual content dataconsumption experience by selectively turning on and off or up and downsettings such as audio volume, audio wideness, color saturation, coloroutput of ambient lighting devices, etc.

Thus, a system along the lines of that shown in FIG. 4 is also capableof influencing the mental state of users of a terminal 2 in accordancewith instructions generated using a remote authoring system 1 andconveyed to the terminal 2 in a data stream over a network 6 or on adata carrier 3.

It should be noted that the above-mentioned embodiments illustrate,rather than limit, the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims. In the claims, any reference signsplaced between parentheses shall not be construed as limiting the claim.The word “comprising” does not exclude the presence of elements or stepsother than those listed in a claim. The word “a” or “an” preceding anelement does not exclude the presence of a plurality of such elements.The mere fact that certain measures are recited in mutually differentdependent claims does not indicate that a combination of these measurescannot be used to advantage.

It will be appreciated that the emotion scene composition segments andsensory scene description segments can be stored as objects on the datacarrier 3. The format of this data is compatible with that of datacurrently in use to provide sub-titles or graphics in data storagesystems (DVD, Blu-Ray) and MPEG-related systems. Indeed, in anotherembodiment, the emotion data can be implemented as a graphics stream orsub-title stream and decoded by applying a firmware update to aconventional terminal 2. Alternatively, the sensory scene descriptionobjects can be described as image data and the emotion compositionobjects can be in the form of composition objects that provideannotations of the image data. A color lookup table can be applied totranslate colors into commands for controlling an ambient output device21,22.

1. Method of adapting an environment of a terminal (2), including:receiving data (27) from an external source at the terminal (2);receiving monitoring data from a system for monitoring at least one userin the environment; and controlling at least one output device forproviding an output perceptible in the environment, the output beingadjusted in dependence on the monitoring data, wherein at least onesegment including data pertaining to at least one mental state isretrieved from the data (27) from the external source, and theadjustment in dependence on the monitoring data is effected at leastpartly on the basis of the data pertaining to the at least one mentalstate.
 2. Method according to claim 1, wherein the data pertaining to atleast one mental state includes data carrying information from which thedependency between the monitoring data and the output is at least partlyderivable.
 3. Method according to claim 1, wherein the data pertainingto at least one mental state includes data representative of a boundaryvalue of an allowable range of values of at least one signal obtainablefrom the monitoring data, and wherein the adjustment of the output iscarried out such as to keep values of the signal obtainable from themonitoring data within the allowable range of values.
 4. Methodaccording to claim 1, wherein segments including data pertaining to atleast one mental state are received in synchrony with content data forrendering in at least one of a visible and an audible manner at theterminal (2), wherein the adjustment of the output perceptible in theenvironment is synchronized with a rendering of the content data at theterminal.
 5. Method according to claim 1, including receiving at leastone segment of the data (27) from the external source including datarepresentative of aspects of the perceptible output to be adjusted, andmodifying the data representative of aspects of the perceptible outputto be adjusted based on the data pertaining to the at least one mentalstate in dependence on the monitoring data.
 6. Method according to claim5, wherein the data representative of aspects of the perceptible outputto be adjusted is further modified on the basis of data representativeof capabilities of at least one of the terminal and the output devicescontrollable by the terminal (2).
 7. Method according to claim 5,including receiving data (27) from an external source including bothsegments including the data representative of aspects of the perceptibleoutput to be adjusted and segments including the data pertaining to theat least one mental state, and separating the respective segments fromeach other.
 8. Method according to claim 7, wherein the data (27) fromthe external source comprises a data stream, and the separation iseffected by demultiplexing the data stream to retrieve separatelyidentifiable data packets including the data representative of aspectsof the perceptible output to be adjusted and data packets including thedata pertaining to the at least one mental state.
 9. System for adaptingan environment of a terminal (2), including: an interface for receivingdata (27) from an external source at the terminal (2); an interface forreceiving monitoring data from a system for monitoring at least one userin the environment; and a device for controlling at least one outputdevice for providing an output perceptible in the environment, whereinthe system is configured to cause the output to be adjusted independence on the monitoring data, and wherein the system is configuredto retrieve from the data from the external source at least one segmentincluding data pertaining to at least one mental state, and to effectthe adjustment in dependence on the monitoring data on the basis of thedata pertaining to the at least one mental state.
 10. (canceled) 11.Method of influencing an environment of a terminal (2), including:providing data (27) to the terminal (2), the terminal (2) being arrangedto receive monitoring data from a system for monitoring at least oneuser in the environment, wherein the terminal (2) is arranged to controlat least one output device for providing an output perceptible in theenvironment, the output being adjusted in dependence on the monitoringdata, and wherein data including at least one segment including datapertaining to a mental state is provided to a terminal (2) arranged toeffect the adjustment in dependence on the monitoring data at leastpartly on the basis of the data pertaining to the mental state. 12.Method according to claim 11, wherein the data pertaining to at leastone mental state includes data carrying information from which thedependency between the monitoring data and the output is at least partlyderivable.
 13. System for influencing an environment of a terminal,configured to provide data (27) for delivery to the terminal (2), theterminal (2) being arranged to receive monitoring data from a system formonitoring at least one user in the environment, wherein the terminal(2) is further arranged to control at least one output device forproviding an output perceptible in the environment, the output beingadjusted in dependence on the monitoring data, and wherein the system isconfigured to provide data (27) including at least one segment includingdata on the basis of which the terminal (2) is able to adjust the outputin dependence on the monitoring data.
 14. Signal carrying a data stream(27) including at least one segment including data pertaining to amental state, which data includes information on the basis of which aterminal (2) arranged to control at least one output device forproviding an output perceptible in an environment of the terminal isable to adjust the output in dependence on monitoring data provided by asystem for monitoring at least one user in an environment of theterminal (2).
 15. Computer program including a set of instructionscapable, when incorporated in a machine-readable medium, of causing asystem having information processing capabilities to perform a methodaccording to claim 1.